Skip to content

test: Strengthen assertions and isolate external HTTP calls - #985

Draft
marandaneto wants to merge 2 commits into
mainfrom
test-audit
Draft

marandaneto wants to merge 2 commits into
mainfrom
test-audit

Conversation

@marandaneto

@marandaneto marandaneto commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

💡 Motivation and Context

Several tests could pass without exercising the behavior they claimed to protect. Some also depended on external HTTP services, scheduler timing, or assertions raised inside worker callbacks that catch exceptions.

This test-only change replaces those HTTP calls with controlled responses and strengthens checks for retry counts, event payloads, exception propagation, truncation, copy isolation, and lifecycle behavior. Consumer tests now wait for delivery and stop their workers. A controlled-clock test checks the remaining flush interval. Registry key checks catch missing or untested OpenAI resources, and placeholder checks retain both explicit expectations and coverage of every registered entry. LangChain error tests still exercise the real callback lifecycle through a mocked HTTP transport.

These are the validated repairs from a partial test audit. This PR does not claim that every repository test has been reviewed. No production code or public API changes are included.

💚 How did you test it?

  • Root suite with external socket connections blocked: 4,143 passed, 16 skipped. The same guarded baseline had 4,137 passed and 6 failures involving external HTTP dependencies.
  • Runtime branch coverage, excluding tests: combined coverage changed from 88.18059% to 88.18510%. Covered lines increased from 14,622 to 14,623 of 16,234. Covered branches stayed at 4,910 of 5,916. The main benefit is stronger assertions rather than broader execution coverage.
  • OpenFeature: 68 passed. Django middleware and exception integration: 7 passed. Compliance adapter: 23 passed.
  • Ruff lint and formatting, mypy baseline filtering, the public API checker tests, and import with warnings treated as errors passed.
  • Changing the tracing default flush interval to 50 seconds in memory passed the old defaults test and failed the repaired test.
  • Autoreview against origin/main reported no actionable findings for 4544aa5.

The full-suite and coverage results above were measured at 107b3fb. After review follow-ups at 4544aa5, 53 focused consumer and OpenAI resource tests, the API checker tests, and Ruff passed. An early-flush mutation passed the previous timing test and failed the new controlled-clock test. Extra and missing wrapper keys were both rejected.

Validation used Python 3.13.13. Live credential-dependent tests, MCP v2, and the full Python-version matrix were not run.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Pi used read-only subagents for test inventories and review, then applied and validated focused test repairs in a dedicated worktree. Tools included Git, pytest, coverage.py, Ruff, mypy, and the autoreview helper. The audit remains incomplete, and unresolved review scope was not treated as approved. Local audit notes are kept outside the PR. No shared session link is available.

@marandaneto marandaneto self-assigned this Sep 26, 2026
@marandaneto

Copy link
Copy Markdown
Member Author

Test coverage comparison

Measured on Python 3.13.13 with the same branch-coverage command before and after. Coverage includes runtime code under posthog/ and excludes posthog/test/*. External socket connections were blocked, with loopback allowed.

Metric Before After Change
Line coverage 90.07022% 90.07638% +0.00616 percentage points
Covered lines 14,622 / 16,234 14,623 / 16,234 +1
Branch coverage 82.99527% 82.99527% Unchanged
Covered branches 4,910 / 5,916 4,910 / 5,916 Unchanged
Combined coverage 88.18059% 88.18510% +0.00451 percentage points
Tests passed 4,137 4,143 +6
Tests failed 6 0 -6
Tests skipped 16 16 Unchanged

The six baseline failures involved external HTTP dependencies under the network guard. The repaired tests use controlled transport responses instead.

The main improvement is stronger assertions and test isolation, not broader execution coverage. As one check, changing the tracing default flush interval to 50 seconds in memory passed the old defaults test but failed the repaired test.

These numbers cover the root suite only. Separate validation passed for OpenFeature (68 tests), Django middleware and exception capture (7 tests), and the compliance adapter (23 tests). The comprehensive semantic audit remains incomplete. Live credential-dependent tests, MCP v2, and the full Python-version matrix were not run.

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

posthog-python Compliance Report

Date: 2026-09-27T14:28:43.255747+00:00
Duration: 259099ms

✅ All Tests Passed!

121/121 tests passed


Capture_V1 Tests

✅ 95/95 tests passed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint ✅ 513ms
Endpoint And Method.Does Not Use Legacy Endpoints ✅ 509ms
Required Headers.Has Authorization Bearer Header ✅ 507ms
Required Headers.Has Content Type Json ✅ 508ms
Required Headers.Has Posthog Sdk Info Format ✅ 508ms
Required Headers.Has Posthog Attempt Header ✅ 507ms
Required Headers.Has Posthog Request Id ✅ 508ms
Required Headers.Has Posthog Request Timestamp ✅ 507ms
Required Headers.Has User Agent ✅ 507ms
Body Format.Body Has Created At And Batch ✅ 508ms
Body Format.No Api Key In Body ✅ 507ms
Body Format.No Sent At In Body ✅ 508ms
Event Format.Event Has Required Root Fields ✅ 507ms
Event Format.Event Uuid Is Valid ✅ 507ms
Event Format.Event Timestamp Is Rfc3339 ✅ 508ms
Event Format.Non Utc Event Timestamp Is Converted To Utc ✅ 511ms
Event Format.Distinct Id Is String ✅ 507ms
Event Format.Distinct Id At Root Not Properties ✅ 508ms
Event Format.Custom Properties Preserved ✅ 507ms
Event Format.Set Properties Preserved ✅ 508ms
Event Format.Set Once Properties Preserved ✅ 508ms
Event Format.Groups Properties Preserved ✅ 507ms
Event Format.Sdk Generates Uuid If Not Provided ✅ 508ms
Event Format.Event Has Required Root Fields Batch ✅ 509ms
Event Format.Event Uuid Is Valid Batch ✅ 510ms
Event Format.Event Timestamp Is Rfc3339 Batch ✅ 510ms
Event Format.Distinct Id Is String Batch ✅ 510ms
Event Format.Distinct Id At Root Not Properties Batch ✅ 510ms
Event Format.Custom Properties Preserved Batch ✅ 509ms
Event Format.Set Properties Preserved Batch ✅ 510ms
Event Format.Set Once Properties Preserved Batch ✅ 510ms
Event Format.Groups Properties Preserved Batch ✅ 510ms
Event Format.Sdk Generates Uuid If Not Provided Batch ✅ 510ms
Batch Behavior.Multiple Events In Single Batch ✅ 513ms
Batch Behavior.Batch Envelope Smoke ✅ 519ms
Batch Behavior.Flush With No Events Sends Nothing ✅ 505ms
Batch Behavior.Flush At Triggers Batch ✅ 1008ms
Batch Behavior.Created At Reflects Batch Creation Time ✅ 508ms
Deduplication.Generates Unique Uuids ✅ 513ms
Deduplication.Different Events Same Content Different Uuids ✅ 509ms
Deduplication.Preserves Uuid On Retry ✅ 6516ms
Deduplication.Preserves Timestamp On Retry ✅ 6512ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 6514ms
Deduplication.No Duplicate Events In Batch ✅ 514ms
Header Behavior On Retry.Attempt Header Starts At One ✅ 507ms
Header Behavior On Retry.Attempt Header Increments On Retry ✅ 13524ms
Header Behavior On Retry.Request Id Preserved On Retry ✅ 6516ms
Header Behavior On Retry.Different Requests Have Different Request Ids ✅ 3015ms
Header Behavior On Retry.Request Timestamp Changes On Retry ✅ 6516ms
Response Format Validation.Success Response Has Uuid Keyed Results ✅ 508ms
Response Format Validation.Success Response Has Ok For Each Event ✅ 510ms
Response Format Validation.Success No Retry After When All Ok ✅ 509ms
Response Format Validation.Success Retry After Present When Retry Events ✅ 1512ms
Response Format Validation.Success No Retry After When Drop Only ✅ 511ms
Response Format Validation.Response Echoes Request Id ✅ 507ms
Retry Behavior.Retries On 408 ✅ 6512ms
Retry Behavior.Retries On 500 ✅ 6516ms
Retry Behavior.Retries On 503 ✅ 8515ms
Retry Behavior.Retries On 504 ✅ 6512ms
Retry Behavior.Retryable Errors Have Retry After ✅ 3514ms
Retry Behavior.Respects Retry After On Retryable Error ✅ 11519ms
Retry Behavior.Does Not Retry On 400 ✅ 2509ms
Retry Behavior.Does Not Retry On 401 ✅ 2509ms
Retry Behavior.Does Not Retry On 402 ✅ 2509ms
Retry Behavior.Does Not Retry On 413 ✅ 2509ms
Retry Behavior.Does Not Retry On 415 ✅ 2509ms
Retry Behavior.Non Retryable Errors Have No Retry After ✅ 2507ms
Retry Behavior.Implements Backoff ✅ 22527ms
Retry Behavior.Max Retries Respected ✅ 22529ms
Partial Batch Handling.Handles 200 Full Success ✅ 2508ms
Partial Batch Handling.Handles 200 With All Ok ✅ 3512ms
Partial Batch Handling.Does Not Retry Dropped Events ✅ 3511ms
Partial Batch Handling.Does Not Retry Limited Events ✅ 3510ms
Partial Batch Handling.Prunes Ok Events On Partial Retry ✅ 6511ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry ✅ 6513ms
Partial Batch Handling.Retries Only Retry Events From Partial ✅ 6517ms
Partial Batch Handling.Partial Retry Preserves Uuids ✅ 6515ms
Partial Batch Handling.Partial Retry Attempt Header Increments ✅ 6514ms
Partial Batch Handling.Partial Retry Request Id Preserved ✅ 6516ms
Partial Batch Handling.Respects Retry After On Partial ✅ 8516ms
Partial Batch Handling.Unknown Result Treated As Terminal ✅ 3509ms
Partial Batch Handling.Mixed Ok Drop Limited No Retry ✅ 3513ms
Compression.Sends Gzip Content Encoding ✅ 506ms
Compression.No Content Encoding When Disabled ✅ 506ms
Compression.Compressed Body Is Decompressible ✅ 506ms
Error Handling.Does Not Retry On Unknown 4Xx ✅ 2509ms
Event Options.Cookieless Mode Override ✅ 505ms
Event Options.Disable Skew Correction Override ✅ 506ms
Event Options.Process Person Profile Override ✅ 507ms
Event Options.Product Tour Id Override ✅ 506ms
Event Options.Unset Options Omitted ✅ 507ms
Event Options.Options Override In Batch ✅ 509ms
Geoip And Historical Migration.Geoip Disable Injected Into Properties ✅ 507ms
Geoip And Historical Migration.Historical Migration Set In Body ✅ 506ms
Geoip And Historical Migration.Historical Migration Absent By Default ✅ 506ms

Capture_Ai Tests

✅ 5/5 tests passed

View Details
Test Status Duration
Routing.Capture Ai Posts To Ai Endpoint ✅ 506ms
Routing.Capture Does Not Reroute Ai Named Events ✅ 506ms
Identity.Capture Ai Event Has Uuid ✅ 506ms
Identity.Capture Ai Keeps Supplied Uuid ✅ 506ms
Timestamp Format.Non Utc Event Timestamp Is Converted To Utc ✅ 507ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 6ms
Request Payload.Flags Request Uses V2 Query Param ✅ 6ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 6ms
Request Payload.Flags Request Omits Authorization Header ✅ 7ms
Request Payload.Token In Flags Body Matches Init ✅ 6ms
Request Payload.Groups Round Trip ✅ 6ms
Request Payload.Groups Default To Empty Object ✅ 5ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 7ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 6ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 6ms
Request Lifecycle.No Flags Request On Init Alone ✅ 2ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 506ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 8ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 6ms
Retry Behavior.Retries Flags On 502 ✅ 308ms
Retry Behavior.Retries Flags On 504 ✅ 308ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 508ms

Feature_Flags_Local_Evaluation Tests

✅ 4/4 tests passed

View Details
Test Status Duration
Versioned Boolean Matching.Matching Version Missing ✅ 43ms
Versioned Boolean Matching.Matching Version 1 ✅ 43ms
Versioned Boolean Matching.Matching Version 2 ✅ 43ms
Versioned Boolean Matching.Version Only Reload 1 2 1 2 Missing ✅ 21ms

@greptile-apps

greptile-apps Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Retrigger

[Low risk] Test improvements and assertion strengthening across test files.

The PR appears safe to merge, with non-blocking gaps in tests intended to protect timing and registry coverage.

Reviews (1) · Last reviewed commit: "test: Strengthen assertions and isolate ..."

Comment thread posthog/test/test_consumer.py
Comment thread posthog/test/ai/openai/test_resource_wrapping.py
Comment thread .github/scripts/test_check_public_api.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant